Image processing apparatus for reducing influence of fine particle in an image, control method of same, and non-transitory computer-readable storage medium

ABSTRACT

This disclosure provides an image processing apparatus comprising a first setting unit which sets a first parameter for processing for removing an influence of a fine particle component based on image data; a first image processing unit which performs fine particle removal processing based on the first parameter; a second setting unit which sets a second parameter; a second image processing unit which performs fine particle removal processing based on the second parameter; a setting unit which sets a region for which the first image processing unit is to be used and a region for which the second image processing unit is to be used; and a generation unit which generates image data by applying a result from the first image processing unit and a result from the second image processing unit to the respective set regions.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing apparatus, controlmethod of same and non-transitory computer-readable storage medium.

Description of the Related Art

In the field of surveillance cameras, etc., the degradation of imagequality of a captured image due to a decrease in visibility caused by afine particle component (for example, fog) present between a camera anda photographic subject is a problem. US-2011-0188775 (referred tohereinafter as document 1) is proposed as a technique (fog/haze removaltechnique) for correcting an image having decreased visibility due tofog, etc. In document 1, visibility is improved by calculating, for eachtarget pixel, the minimum value of the R, G, and B channels within apredetermined range around the target pixel, and correcting the contrastusing the minimum value image. Furthermore, in US-2016-0328832 (referredto hereinafter as document 2), a histogram is calculated from an inputimage, and parameters for fog/haze removal processing are determinedbased on the likelihood and the kurtosis of the histogram.

In the technique disclosed in document 1, parameters used during imageprocessing are uniquely held with respect to the entire image. However,there are cases in which it is better to vary parameters between a casein which the fog/haze removal technique is applied to a photographicsubject at a relatively close distance and a case in which the fog/hazeremoval technique is applied to a photographic subject at a fartherdistance. Here, there is a possibility that, if processing is performedon a photographic subject at a close distance using parameters for aphotographic subject at a farther distance, the image will be unnaturaldue to the effect of the processing being too strong for thephotographic subject at a close distance.

Furthermore, in the technique disclosed in document 2, parameters aredetermined based on the histogram of an entire image. Due to this, evenif a user would like to improve the visibility of a specific object (aperson, for example) in an image, processing is executed so as toimprove the visibility over the entire image, and the visibility of aperson present at a position where fog is thick, in particular, is notimproved. Furthermore, there is a possibility that, if an attempt ismade to improve the visibility of a person present at a position wherefog is thick, the image will be unnatural due to the fog/haze removalprocessing being excessively applied to a person whose visibility isalready secured.

SUMMARY OF THE INVENTION

The present invention provides a technique for removing the influence offog and haze in an image in accordance with the conditions of theshooting scene and an object whose visibility is desired to be improved.

According to a first aspect of the invention, there is provided an imageprocessing apparatus that processes image data obtained by imagecapturing performed by an image-capturing unit, the image processingapparatus comprising: a first setting unit configured to set a firstparameter for performing processing for removing an influence of a fineparticle component based on the image data; a first image processingunit configured to perform fine particle removal processing based on thefirst parameter set by the first setting unit; a second setting unitconfigured to set a second parameter differing from the first parameter;a second image processing unit configured to perform fine particleremoval processing based on the second parameter set by the secondsetting unit; a third setting unit configured to set a region for whichthe first image processing unit is to be used and a region for which thesecond image processing unit is to be used; and a generation unitconfigured to generate image data from which an influence of fineparticles is removed by applying a result from the first imageprocessing unit and a result from the second image processing unit tothe respective regions set by the third setting unit.

According to a second aspect of the invention, there is provided amethod of controlling an image processing apparatus that processes imagedata obtained by image capturing performed by an image-capturing unit,the method comprising: a first setting step of setting a first parameterfor performing processing for removing an influence of a fine particlecomponent based on the image data; a first image processing step ofperforming fine particle removal processing based on the first parameterset in the first setting step; a second setting step of setting a secondparameter differing from the first parameter; a second image processingstep of performing fine particle removal processing based on the secondparameter set in the second setting step; a third setting step ofsetting a region for which the first image processing step is to be usedand a region for which the second image processing step is to be used;and a generation step of generating image data from which an influenceof fine particles is removed by applying a result from the first imageprocessing unit and a result from the second image processing unit tothe respective regions set by the setting unit.

According to a third aspect of the invention, there is provided anon-transitory computer-readable storage medium storing a programexecutable by a computer to execute a method of controlling an imageprocessing apparatus that processes image data obtained by imagecapturing performed by an image-capturing unit, the method comprising: afirst setting step of setting a first parameter for performingprocessing for removing an influence of a fine particle component basedon the image data; a first image processing step of performing fineparticle removal processing based on the first parameter set in thefirst setting step; a second setting step of setting a second parameterdiffering from the first parameter; a second image processing step ofperforming fine particle removal processing based on the secondparameter set in the second setting step; a third setting step ofsetting a region for which the first image processing step is to be usedand a region for which the second image processing step is to be used;and a generation step of generating image data from which an influenceof fine particles is removed by applying a result from the first imageprocessing unit and a result from the second image processing unit tothe respective regions set by the setting unit.

According to the present invention, the influence of fog and haze in animage is removed in accordance with the conditions of the shooting sceneand an object whose visibility is desired to be improved.

Further features of the present invention will become apparent from thefollowing description of an exemplary embodiment (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block configuration diagram of an image processing apparatusin an embodiment.

FIG. 2 is a functional block diagram of the image processing apparatusdescribed in the embodiment.

FIG. 3 is a diagram illustrating an internal configuration of a fineparticle removal processing unit described in the embodiment.

FIG. 4 is a flowchart illustrating processing in the image processingapparatus according to the embodiment.

FIG. 5 is a flowchart illustrating fine particle removal processingaccording to the embodiment.

FIGS. 6A and 6B are schematic diagrams illustrating the process oflower-pixel image generation according to the embodiment.

FIG. 7 is a flowchart illustrating airglow estimation processingaccording to the embodiment.

FIG. 8 is a flowchart illustrating lower-pixel corrected imagegeneration processing according to the embodiment.

FIG. 9 is a flowchart illustrating processing for generating an RGBlower-pixel-value corrected image according to the embodiment.

FIGS. 10A to 10D are schematic diagrams illustrating filter processingin RGB lower-pixel image generation processing according to theembodiment.

FIG. 11 is a flowchart illustrating Mie scattering component generationprocessing according to the embodiment.

FIG. 12 is a flowchart illustrating Rayleigh scattering componentgeneration processing according to the embodiment.

DESCRIPTION OF THE EMBODIMENT

Hereinafter, embodiments will be described in detail with reference tothe attached drawings. Note, the following embodiments are not intendedto limit the scope of the claimed invention. Multiple features aredescribed in the embodiments, but limitation is not made an inventionthat requires all such features, and multiple such features may becombined as appropriate. Furthermore, in the attached drawings, the samereference numerals are given to the same or similar configurations, andredundant description thereof is omitted.

Overview

In the present embodiment, processing for removing the influence of afine particle component (referred to hereinafter as fine particles),such as fog, is first performed on an input image shot under conditionsin which the fine particle component is generated. Next, an object suchas a person is extracted from the image from which the influence of thefine particle component has been removed, by performing object detectionprocessing such as person detection. By changing the ratio between theMie scattering component and the Rayleigh scattering component in theinput image based on this extraction result when generating a fineparticle removal image, images in which the appearance of the influenceof the fine particle component is varied are created. An image in whichthe object has higher visibility is obtained by combining these images.

Device Configuration According to Embodiment

FIG. 1 is a block configuration diagram of an image processing apparatus100 to which the present embodiment applies. The image processingapparatus 100 according to the present embodiment includes a CPU 101, aRAM 102, a ROM 103, an HDD interface (I/F) 104, an HDD 105, an input I/F106, an output I/F 107, and a system bus 108. The CPU 101 is a processorthat performs overall control of the constituent units described below.The RAM 102 is a memory that functions as the main memory and work areaof the CPU 101. The ROM 103 is a memory that stores various parametersand a program for controlling processing in the image processingapparatus 100.

The HDD I/F 104 is an interface conforming to the Serial ATA (SATA)standard, etc., for example, and connects the HDD 105, which serves as asecondary storage apparatus, to the system bus 108. The CPU 101 can readdata from the HDD 105 and write data to the HDD 105 via the HDD I/F 104.Furthermore, the CPU 101 can load data stored in the HDD 105 into theRAM 102, and can similarly store data loaded in the RAM 102 to the HDD105. Also, the CPU 101 can execute data loaded into the RAM 102,regarding the data as a program. Note that the secondary storageapparatus may be a storage device other than a HDD, such as an opticaldisk drive. The input I/F 106 is a serial bus interface conforming tothe USB standard, the IEEE1394 standard, etc., for example.

The image processing apparatus 100 is connected to an external memory109 and an image-capturing unit 111 via the input I/F 106. The CPU 101can obtain captured image data from the external memory 109 and theimage-capturing unit 111 via the input I/F 106. The output I/F 107 is avideo output interface conforming to the DVI standard, the HDMI(registered trademark) standard, etc., for example. The image processingapparatus 100 is connected to a display unit 110 via the output I/F 107.The CPU 101 can display images on the display unit 110 by outputting theimages to the display unit 110 via the output I/F 107.

The system bus 108 is a transfer path for various types of data, and theconstituent units in the image processing apparatus 100 are connected toone another via the system bus 108.

The external memory 109 is a storage medium such as a hard disk, amemory card, a CF card, an SD card, or a USB memory, and can store datasuch as image data processed by the image processing apparatus 100.

The display unit 110 is a display apparatus such as a display, and candisplay images processed by the image processing apparatus 100, etc.

The image-capturing unit 111 is a camera that uses an image sensor toreceive an optical image of a photographic subject and outputs theobtained optical image as digital image data. In the image processingapparatus 100 according to the present embodiment, image data whosecontrast is decreased due to scattered light generated by fineparticles, such as those of fog, is obtained by the image-capturing unit111 through image capturing, and the image processing apparatus 100generates an image in which the influence of fine particles is reducedby performing image processing described below.

An operation unit 112 is constituted by one or more input devices suchas a mouse and/or a keyboard, for example, and is used for specifyingthe later-described fog/haze removal range.

FIG. 2 is a functional block diagram of the image processing apparatusdescribed in the embodiment. As illustrated in FIG. 2 , the imageprocessing apparatus according to the embodiment includes an input imagedata obtaining unit 201, a fine particle removal processing unit 202, afine particle removal image data output unit 203, an input image datastoring unit 204, a fine particle removal image data storing unit 205,and an object extraction processing unit 206. The object extractionprocessing unit 206 is constituted by a known person detectiontechnique, a known face detection technique, etc. The object extractionprocessing unit 206 performs detection of a shape of a person, aperson’s face, etc., on an input image, and stores an area correspondingto the shape of a person, a person’s face, etc., in an object detectionresult storing unit 207 as a detection result. Note that, while theprocessing units illustrated in FIG. 2 are realized by the CPU 101loading a program stored in the ROM 103 in the RAM 102 and executing theprogram, some of the processing units may be realized by means ofhardware.

FIG. 3 is a diagram illustrating an internal configuration of the fineparticle removal processing unit 202 in the embodiment. The fineparticle removal processing unit 202 includes an airglow calculatingunit 301, a lower-pixel image calculating unit 302, a lower-pixelimage-based correction processing unit 304 (hereinafter as correctionprocessing unit 304), and an RGB lower-pixel image-based correctionprocessing unit 305 (hereinafter as correction processing unit 305). Thefine particle removal processing unit 202 also includes a Mie scatteringcomponent calculating unit 306 and a Rayleigh scattering componentcalculating unit 307 for controlling scattering components, and acomposing unit 308. Furthermore, the fine particle removal processingunit 202 includes, as storage locations of data for these various typesof processing, an airglow data storing unit 309, a lower-pixel imagedata storing unit 310, a lower-pixel corrected data storing unit 312,and an RGB lower-pixel corrected data storing unit 313. The fineparticle removal processing unit 202 includes a Mie scattering componentdata storing unit 314 and a Rayleigh scattering component data storingunit 315 for controlling scattering components. Furthermore, the fineparticle removal processing unit 202 includes an image processing rangedata storing unit 316 that determines a processing range for fineparticle removal processing.

While these constituent blocks are realized by the CPU 101 executingprograms held in the ROM 103, the HDD 105, and the RAM 102, which serveas data holding areas, as necessary, some of the constituent blocks maybe realized by means of hardware.

Overall Processing Flow

The flow of processing in the image processing apparatus 100 accordingto the embodiment will be described using the block diagram in FIG. 2and the flowchart in FIG. 4 .

In step S401, the CPU 101 controls the input image data obtaining unit201 and causes the input image data obtaining unit 201 to obtain imagedata obtained through image capturing by the image-capturing unit 111,and stores the image data in the input image data storing unit 204.

In step S402, the CPU 101 sets a parameter for performing processing forremoving a fine particle component on the input image data.

In step S403, the CPU 101 performs processing (described in detaillater) for removing the influence of fine particles, based on theparameter set in step S402.

In step S404, the CPU 101, by using known object detection processing,performs object detection on the image subjected to the fine particleremoval processing in step S403. Here, the objects to be detected areobjects whose visibility the user would like to improve. For example, ifthe user would like to improve the visibility of a person, the CPU 101extracts a person in the image data by applying known person detectionprocessing. The CPU 101 encloses, with a rectangle, the surroundingregion of a person area resulting from the extraction, and stores theperson area in the form of position information of the rectangle area inan object detection result storing unit 207. Note that, while an objectdetection result is a rectangle area in the present embodiment, theshape of the object detection result is not particularly limited, andthe object detection result may have a shape other than a rectangularshape. Note that, in a case in which an object detection result is not arectangle area, for example, it suffices to determine, for each pixel,whether the pixel is a pixel in which an object was detected.

In step S405, the CPU 101 compares the image subjected to the processingfor removing the influence of fine particles and the object detectionresult, determines which region in the image data an object was detectedin, and varies processing depending upon the result of thedetermination. Specifically, the CPU 101 shifts to the processing instep S408 for pixels corresponding to a region for which it has beendetermined that an object was detected in the image data, and shifts tothe processing in step S406 for pixels corresponding to a region forwhich it has been determined that no object was detected.

In step S406, the CPU 101 sets a parameter for the fine particle removalprocessing to be executed in the subsequent step S407 so that the objectdetection accuracy is further increased for the region in which noobject was detected. Specifically, the parameter is set so that the fineparticle removal effect is increased. For example, the parameter is setso that the later-described Mie scattering intensity coefficient m andthe later-described Rayleigh scattering intensity coefficient r aresmaller in the parameter for the second iteration than in the parameterfor the first iteration. For m in particular, setting m in the parameterfor the second iteration to be smaller, setting m in the parameter forthe second iteration to zero, etc., can be considered.

In step S407, the CPU 101 performs the fine particle removal processingonce again based on the parameter set in step S406, which is for thesecond iteration of the fine particle removal processing. The image datathat is processed here is not the image data subjected to the processingin step S403, and is the original input image data obtained in stepS401.

Then, in step S408, the CPU 101 combines the image that is the result ofthe processing using the first parameter, for which it has beendetermined in step S405 that an object was detected, and the image thatis the result of the processing using the second parameter performed instep S407. Specifically, one output image is generated using, for thedetection region (pixels), which is the region for which it has beendetermined in step S405 that an object was detected, the image that isthe result of the processing using the first parameter, and using, forthe non-detection region, the image that is the result of the processingusing the second parameter.

In such a manner, regions are set based on the object detection result,and for each of the regions, an image subjected to fine particlecomponent removal processing having a different effect is generated.

Flow of Processing for Removing Influence of Fine Particles

Here, the fine particle removal processing in steps S403 and S407 ofFIG. 4 will be described in detail using the processing block diagramsin FIGS. 2 and 3 and the flowchart in FIG. 5 . Note that step S403 andstep S407 differ in that the parameter used for the processing differs.

In step S501, the lower-pixel image calculating unit 302 calculates alower-pixel image (described in detail later) from the input image datastored in the input image data storing unit 204, and stores thelower-pixel image in the lower-pixel image data storing unit 310.

In step S502, the airglow calculating unit 301 calculates an airglowcomponent (described in detail later) using the input image data storedin the input image data storing unit 204 and the lower-pixel image datastored in the lower-pixel image data storing unit 310. Then, the airglowcalculating unit 301 stores the calculated airglow data in the airglowdata storing unit 309.

In step S503, the lower-pixel image-based correction processing unit 304reads the airglow data stored in the airglow data storing unit 309 andthe lower-pixel image data stored in the lower-pixel image data storingunit 310. Furthermore, the correction processing unit 304 also readsimage processing range data stored in the image processing range datastoring unit 316. Then, the correction processing unit 304 performscorrection (described in detail later) on the input image data stored inthe input image data storing unit 204. The correction processing unit304 stores the corrected image data in the lower-pixel corrected datastoring unit 312.

In step S504, the RGB lower-pixel image-based correction processing unit305 reads the airglow data stored in the airglow data storing unit 309,the input image stored in the input image data storing unit 204, and theimage processing range data stored in the image processing range datastoring unit 316. Then, this correction processing unit 305 performscorrection (described in detail later) on the input image data. Thecorrection processing unit 305 stores the corrected image data in theRGB lower-pixel corrected data storing unit 313.

In step S505, the Mie scattering component calculating unit 306 readsthe input image data stored in the input image data storing unit 204 andthe lower-pixel corrected image data stored in the lower-pixel correcteddata storing unit 312. Then, the Mie scattering component calculatingunit 306 calculates the Mie scattering component (described in detaillater). The Mie scattering component calculating unit 306 stores thecalculated Mie scattering component data in the Mie scattering componentdata storing unit 314.

In step S506, the Rayleigh scattering component calculating unit 307reads the input image data stored in the input image data storing unit204. Furthermore, the Rayleigh scattering component calculating unit 307also reads the lower-pixel corrected image data stored in thelower-pixel corrected data storing unit 312 and the RGBlower-pixel-value corrected image data stored in the RGB lower-pixelcorrected data storing unit 313. Then, the Rayleigh scattering componentcalculating unit 307 calculates the Rayleigh scattering component(described in detail later), and stores the calculated Rayleighscattering component in the Rayleigh scattering component data storingunit 315.

In step S507, the composing unit 308 reads the RGB lower-pixel-valuecorrected image data stored in the RGB lower-pixel corrected datastoring unit 313. Furthermore, the composing unit 308 reads the Miescattering component data stored in the Mie scattering component datastoring unit 314 and the Rayleigh scattering component data stored inthe Rayleigh scattering component data storing unit 315. Subsequently,the composing unit 308 performs image composition (described in detaillater), and stores the composed image data in the fine particle removalimage data storing unit 205.

As a result of the processing described above, the fine particle removalprocessing in step S403 is completed.

Airglow Estimation

Next, the airglow calculation processing in step S502 will be described.The airglow calculating unit 301 first converts the input image from anRGB image into a luminance image (Y image). Next, the airglowcalculating unit 301 generates a histogram from the Y image obtained asa result of the conversion, sets a value corresponding to the top 1% asa threshold, and performs robust estimation processing to determinepixels for estimating the airglow from among positions of pixels havingpixel values greater than or equal to the threshold. Furthermore, theairglow calculating unit 301 estimates the airglow based on the pixelvalues of the determined pixels.

In the following, the details of the airglow calculating unit 301 in theembodiment will be described with reference to the flowchart in FIG. 7 .

In step S701, the airglow calculating unit 301 reads the input imagedata from the input image data storing unit 204.

In step S702, the airglow calculating unit 301 converts the input imagedata read from an RGB image into a Y image. Here, a conventional formulafor color conversion from RGB into Y may be applied as the conversionformula.

In step S703, the airglow calculating unit 301, from the Y image(luminance image) obtained through the conversion in step S702,generates candidates (referred to hereinafter as pixel positioncandidates) of airglow position information for performing airglowestimation. Specifically, the airglow calculating unit 301 calculates ahistogram of the read Y image, sets a value corresponding to the top 1%from the maximum value as a threshold, and determines the positions ofpixels having values greater than or equal to the threshold as referencepixel position candidates. Note that, while the top 1% is set as thethreshold in the present embodiment, the embodiment is not limited tothis, and a different percentage may be adopted.

In step S704, the airglow calculating unit 301 determines referencepixel position information (referred to hereinafter as pixel positions)for actually calculating the airglow. Specifically, based on the pixelposition candidates determined in step S703, the airglow calculatingunit 301 generates airglow position information using robust estimationsuch as the RANSAC method, based on the candidates. This is becausepixel positions corresponding to the sky portion are naturally desirableas pixel positions to be selected as airglow, and the exclusion of highluminance portions other than the sky in the image from the pixelposition candidates is desired. Generally, high luminance portions otherthan the sky occupy a small proportion in an image, and tend to have aluminance different from the color of the sky. Thus, robust estimationis performed in which high luminance portions other than the sky can betreated as outliers and excluded. Furthermore, the number of pixelpositions can also be limited in this process. This is for avoiding thefollowing situation; in a case such as when there is a gradation in thecolor of the sky in an image, the same sky in the image includesdifferent pixel values, and thus even a sky portion where the colorchanges would be subjected to estimation if too many pixels thereof arereferred to.

In step S705, in order to calculate the airglow, the airglow calculatingunit 301 determines the pixel position from which the airglow componentis to be extracted first from among the pixel positions determined instep S704. In doing so, it suffices to determine the first pixelposition in the raster scan order (for example, the top-left most pixelposition) from among the pixel positions determined in step S704 as thepixel position from which the airglow component is to be extractedfirst.

In step S706, the airglow calculating unit 301 adds the pixel values (R,G, B) of the reference pixel position initially determined in step S705or determined in step S708 color by color, and holds the results in theRAM 102, etc.

In step S707, the airglow calculating unit 301 determines whether or notthe search has been performed for all pixel positions determined in stepS704. The airglow calculating unit 301 advances the processing to stepS709 if it is determined that the search has been performed for allpixel positions, and advances the processing to step S708 if it isdetermined that the search is not complete.

In step S708, the airglow calculating unit 301 moves the pixel positiondetermined in step S704 to the next position. Specifically, among thepixel positions determined in step S704, the pixel position that isclosest in the raster scan order to the pixel position that is currentlybeing referred to is set.

In step S709, the airglow calculating unit 301 calculates the airglowcomponent by averaging the added pixel values added and held in the RAM102, etc., in step S706. Specifically, the airglow calculating unit 301calculates the airglow component A_(RGB) based on the formulas below.

$\begin{matrix}{\text{A}_{\text{RGB}} = \left( {{{\text{Σ}\text{A}_{\text{R}}}/\text{n}},{{\text{Σ}\text{A}_{\text{G}}}/\text{n}},{{\text{Σ}\text{A}_{\text{B}}}/\text{n}}} \right)} & \text{­­­(1)}\end{matrix}$

$\begin{matrix}{\text{A}_{\text{Y}} = {\left( {{{\text{Σ}\text{A}_{\text{R}}}/\text{n}} + {{\text{Σ}\text{A}_{\text{G}}}/\text{n}} + {{\text{Σ}\text{A}_{\text{B}}}/\text{n}}} \right)/3}} & \text{­­­(2)}\end{matrix}$

Here, A_(R), A_(G), A_(B), and A_(Y) respectively indicate the airglowcomponent values of the R channel, the G channel, the B channel, and thelower-pixel image. Furthermore, n indicates the total number ofreference pixels determined in step S704, and Σ indicates the sum of thevalues of the pixels determined in step S704. Note that formulas (1) and(2) given here are merely examples, and a different formula may be usedas a calculation formula for the airglow estimation in the embodiment.For example, formula (2) may be replaced with the smallest value amongΣA_(R)/n, ΣA_(G)/n, and ΣA_(B)/n.

The airglow component can be estimated as described above. The airglowcalculating unit 301 stores the estimated airglow component in theairglow data storing unit 309.

Creation of Lower-Pixel Image

Next, the lower-pixel image generation processing performed by thelower-pixel image calculating unit 302 in step S501 will be describedusing FIGS. 6A and 6B. As illustrated in FIG. 6A, peripheral pixelscentered around a given target pixel P5 in the input image are denotedas pixels P1 to P4 and pixels P6 to P9. Furthermore, the R, G, and Bcomponent values of the pixels P1 to P9 are expressed as P1(R1, G1, B1)to P9(R9, G9, B9).

Furthermore, suppose that these component values are ranked in the orderof R5 > B3 > R2 > ... > R4 > B1 > G9 > G7. Here, when a lower-pixel ofthe target pixel P5 is defined as T1 as illustrated in FIGS. 6B, T1 is aweighted average of the three lower ranking component values excludingthe lowest component value G7, as indicated in formula (3). By adoptinga weighted average rather than the minimum value, a situation in whichthe lower-pixel image is highly influenced by sensor noise can beprevented. That is, a situation in which a pixel that is highlyinfluenced by sensor noise is generated in the processed image can besuppressed compared to the case in which the minimum value is adopted.

$\begin{matrix}{\text{T1} = {\left( {2 \times \text{R4 + 4} \times \text{B1 + 2} \times \text{G9}} \right)/8}} & \text{­­­(3)}\end{matrix}$

The lower-pixel image calculating unit 302 generates the lower-pixelimage by performing the above-described processing for all pixels.Furthermore, the lower-pixel image calculating unit 302 stores thegenerated lower-pixel image in the lower-pixel image data storing unit310. Note that the calculation method mentioned above is an example ofthe calculation formula for calculating the lower-pixel image, andcalculation need not be performed following execution of thiscalculation formula. For example, calculation may be performed byaveraging four lower ranking pixels from the second-to-lowest pixel.Furthermore, while a lower-pixel is generated by referring to peripheralpixels located at a distance of one pixel from the target pixel in thepresent embodiment, a lower-pixel may of course be calculated byreferring to peripheral pixels located at a distance of two pixels, orreference may be made to peripheral pixels at a farther distance. Itshould be understood that FIGS. 6A and 6B are merely examples.

Generation of Corrected Image Based on Lower-Pixel Image

Next, the corrected image generation processing (step S503 in FIG. 5 )based on the lower-pixel image, which is performed by the correctionprocessing unit 304, will be described with reference to the flowchartin FIG. 8 .

In step S801, the correction processing unit 304 reads the lower-pixelimage, the airglow data, and the input image from the airglow datastoring unit 309, the lower-pixel image data storing unit 310, and theinput image data storing unit 204.

In step S802, the correction processing unit 304 calculates a correctedlower-pixel image lower_A by correcting the lower-pixel image using theairglow data. Specifically, the correction processing unit 304 correctsthe lower-pixel image based on the airglow data, according to formula(4) below.

$\begin{matrix}{\text{lower\_A} = {\text{T}_{\text{in\_lower}}/\text{A}_{\text{Y}}}} & \text{­­­(4)}\end{matrix}$

Here, T_(in_) _(lower) indicates the lower-pixel image generated in stepS501.

In step S803, the correction processing unit 304 generates atransmission distribution t_(lower)(x, y) based on the correctedlower-pixel image lower_A calculated in step S802. Specifically, theformula below is applied to lower_A(x, y) generated in step S802.

$\begin{matrix}{\text{t}_{\text{lower}} = 1.0\text{-}\text{ω} \times \text{lower\_A}\left( {\text{x,}\mspace{6mu}\text{y}} \right)} & \text{­­­(5)}\end{matrix}$

Here, ω is a coefficient for adjustment, and is “0.9” for example.Furthermore, x and y are horizontal-direction and vertical-directioncoordinates in the image. The coefficient ω is a value provided in orderto prevent a value of a target pixel subjected to the fine particleremoval processing from equaling zero due to transmission equaling zeroin a case in which the transmission light of the pixel is consisted onlyof light scattered by fine particles, such as those of fog, and need notbe “0.9” as mentioned above.

In step S804, the correction processing unit 304 shapes the transmissiondistribution generated in step S803 in accordance with the input imageand the image processing range data, which is input from a UI. Thisshaping is performed because the transmission distribution t_(lower)(x,y) needs to match the shapes of photographic subjects such as structuresincluded in the image-captured data, and in order to limit theprocessing range to a transmission distribution range specified by meansof the UI. In the processing up to step S803, the transmissiondistribution t(x, y) only includes information regarding approximatephotographic subject shapes in the image-captured data. Thus, theshaping is performed because photographic subject shapes need to beaccurately separated. Specifically, it suffices to use a knownedge-preserving filter as that disclosed in the document “Guided ImageFiltering,” Kaiming He, Jian Sun, and Xiaoou Tang, in ECCV2010 (Oral).

Next, measures are taken so that the processing for removing theinfluence of fine particles is not performed on pixel portions outsidethe transmission distribution range specified using the UI. For valuesexceeding t_th_max and values falling below t_th_min that have beenspecified by means of the UI, t_(lower)(x, y) = 1 is substituted intothe transmission distribution t_(lower)(x, y).

Note that, while the values set by means of the UI are applied after thefilter processing in the present embodiment, the values set by means ofthe UI first may of course be applied.

In step S805, the correction processing unit 304 calculates a correctedimage that is based on the lower-pixel image from A_(Y) and thetransmission distribution t_(lower)(x, y). Specifically, this isperformed based on formula (6) below.

$\begin{matrix}{J_{lower}\left( {x,y} \right) = \frac{\left( {I\left( {x,y} \right) - \text{A}_{\text{Y}}} \right)}{\max\left( {t_{0},\text{t}_{\text{lower}}\left( {x,y} \right)} \right)} + \text{A}_{\text{Y}}} & \text{­­­(6)}\end{matrix}$

Here, J_(lower) is the corrected image based on the lower-pixel image, Iis the input image, and t₀ is a coefficient for adjustment, and is “0.1”for example. Here, t₀ is a value provided in order to prevent asituation in which the value of J_(lower) fluctuates significantly dueto a slight difference from the input image I, such as shot noise duringthe image capturing, in a case in which t_(lower) is an extremely smallvalue, and need not be “0.1” as mentioned above. Further, max(···) is afunction that returns the maximum value of the group of numerical valueslined up inside the brackets.

In step S806, the correction processing unit 304 stores the lower-pixelcorrected image J_(lower) calculated in step S805 in the lower-pixelcorrected data storing unit 312.

By performing the above-described processing, an image which is based onthe lower-pixel image and in which the influence of the fine particlecomponent is corrected can be created.

Generation of Corrected Image Based on RGB Lower-Pixel-Value Image

Next, the corrected image generation processing (step S504 in FIG. 5 )performed by the RGB lower-pixel image-based correction processing unit305 will be described with reference to the flowchart of FIG. 9 .

In step S901, the correction processing unit 305 reads the input imagefrom the input image data storing unit 204, and reads the airglow datafrom the airglow data storing unit 309.

In step S902, the correction processing unit 305 calculates an RGBlower-pixel-value image patch_RGB_A corrected using the airglow, byperforming correction (filter processing) on the input image for each ofthe planes R, G, and B using the airglow data.

First, the correction processing unit 305 calculates the RGB lower-pixelimage patch_RGB_A corrected using the airglow by correcting an RGBlower-pixel-value image using the airglow data, according to formula (7)below.

$\begin{matrix}{\text{RGB\_A}\left( \text{x, y, c} \right) = {{\text{T}_{\text{in\_RGB}}\left( \text{x, y, c} \right)}/\text{A}_{\text{RGB}}}} & \text{­­­(7)}\end{matrix}$

Here, T_(in) _(_RGB) indicates the RGB lower-pixel-value image databefore correction, and RGB_A indicates the corrected RGBlower-pixel-value image data. Further, x and y indicatehorizontal-direction and vertical-direction coordinates in the image,and c indicates a color plane (which is either R, G, or B).

Next, the correction processing unit 305 calculates the RGB lower-pixelimage patch_RGB_A corrected using the airglow by performing filterprocessing on the previously-calculated RGB_A. In the processingfollowing step S903, patch_RGB_A is used for all calculations as the RGBlower-pixel image corrected using the airglow.

Here, details of the filter processing in the present processing will bedescribed with reference to FIGS. 10A to 10D.

In FIGS. 10A to 10D, the process of performing filter processing on agiven target pixel T3 is illustrated as schematic diagrams. FIG. 10Aillustrates RGB_A. In FIGS. 10B to 10D, FIG. 10A is shown separated intothe planes R, G, and B. T3 indicates the target pixel to be processed inthe filtering, and T3_(R), T3_(G), and T3_(B) respectively indicate thecomponent value of the target pixel T3 in the planes R, G, and B.Furthermore, R1 to R4, G1 to G4, and B1 to B4 in FIGS. 10B to 10Dindicate four values counted from the minimum value within a range of 5x 5 pixels from the target pixel in the planes R, G, and B,respectively, and the pixel values are ranked R4 > R3 > R2 > R1 in orderof greater value. G1 to G4 for the G component and B1 to B4 for the Bcomponent have the same meanings.

In a case in which filter processing is performed on the corrected RGBlower-pixel-value image data RGB_A, the lower-pixels of the planes R, G,and B within the range of 5 × 5 pixels with the target pixel T3 at thecenter differ for each color, as illustrated in FIGS. 10B to 10D. Due tothis, a lower-pixel filter processing result T3_(R) for the R channel asillustrated in FIG. 10B is calculated according to formula (8) below.

$\begin{matrix}{\text{T3}_{\text{R}} = {\left( {2 \times \text{R2} + 4 \times \text{R3} + 2 \times \text{R4}} \right)/8}} & \text{­­­(8)}\end{matrix}$

Similarly, the result obtained by substituting G2, G3, and G4 intoformula (8) above will be the result in a case in which the minimumvalue T3_(G) for the G channel is calculated. This similarly applies tothe minimum value T3_(B) for the B channel as well. The difference fromthe lower-pixel image is that, for these values, pixel values of thesame plane as the plane of the target pixel are adopted. The lower-pixelimage differs from the RGB lower-pixel image, and calculation isperformed using the pixels of all of the planes around the target pixel.Thus, pixels from any of the planes R, G, and B may be adopted, but inthe case of the RGB lower-pixel-value, pixels are adopted from only thesame plane. Due to this difference, the influence of wavelengths ofscattered light can be taken into account.

Following this, the RGB lower-pixel-value image patch_RGB_A correctedusing the airglow is calculated by applying this processing to allpixels of RGB_A.

In step S903, the correction processing unit 305 creates a transmissiondistribution t_(RGB)(x, y, c) based on the RGB lower-pixel-value imagecorrected using the airglow, which was calculated in step S902. Thefollowing formula is applied to patch_RGB_A generated in step S902.

$\begin{matrix}{\text{t}_{\text{RGB}} = 1.0\text{-}\text{ω} \times \text{patch\_RGB\_A}\left( {\text{x,}\mspace{6mu}\text{y,}\mspace{6mu}\text{c}} \right)} & \text{­­­(9)}\end{matrix}$

Here, ω is a coefficient for adjustment, and is for example 0.9. ω is avalue provided in order to prevent a value of a target pixel subjectedto fine particle removal processing from equaling zero due totransmission equaling zero in a case in which the transmission light ofthe pixel is consisted only of light scattered by fine particles, suchas those of fog, and need not be 0.9 as mentioned above.

In step S904, the correction processing unit 305 shapes the transmissiondistribution generated in step S903 in accordance with the input image,and ensures that processing is not performed on portions outside thetransmission distribution range specified by means of the UI. Thespecific procedures are the same as those in step S804, but in the caseof the RGB lower-pixel-value image, the processing in step S804 isperformed for each color plane of the transmission distributiont_(RGB)(x, y, c).

In step S905, the correction processing unit 305 calculates a correctedimage based on the RGB lower-pixel-value image from the airglow A_(RGB)and the transmission distribution t_(RGB)(x, y, c). Specifically, thisis performed based on formula (10) below.

$\begin{matrix}{J_{RGB}\left( {x,y,c} \right) = \frac{\left( {I\left( {x,y,c} \right) - \text{A}_{\text{RGB}}} \right)}{\max\left( {t_{0},\text{t}_{\text{RGB}}\left( {x,y,c} \right)} \right)} + \text{A}_{\text{RGB}}} & \text{­­­(10)}\end{matrix}$

Here, J_(RGB) is the corrected image based on the RGB lower-pixel-valueimage, and I is the input image. Furthermore, t₀ is a coefficient foradjustment, and is 0.1, for example. Here, t₀ is a value provided inorder to prevent a situation in which the value of J_(RGB) fluctuatessignificantly due to a slight difference from the input image I, such asshot noise during the image capturing, in a case in which t_(RGB) is anextremely small value, and need not be 0.1 as mentioned above.

In step S906, the correction processing unit 305 stores J_(RGB)calculated in step S905 in the RGB lower-pixel corrected data storageunit 313.

By performing the above-described processing, an image obtained bycorrecting the RGB lower-pixel-value image can be created.

Calculation of Mie Scattering Component

Next, processing (step S505) for calculating a light scatteringcomponent deriving from Mie scattering from the lower-pixel correctedimage data and the input image, which is performed by the Mie scatteringcomponent calculating unit 306, will be described with reference to theflowchart in FIG. 11 .

In step S1101, the Mie scattering component calculating unit 306 readsthe lower-pixel corrected data from the lower-pixel corrected datastoring unit 312, and reads the input image from the input image datastoring unit 204.

In step S1102, the Mie scattering component calculating unit 306subtracts a pixel value for each pixel in the image to extract the Miescattering component. Specifically, the Mie scattering componentcalculating unit 306 calculates a Mie scattering component imageaccording to formula (11) below.

$\begin{matrix}\begin{array}{l}{\cdot \text{When I}\left( \text{x, y, c} \right)\text{- J}_{\text{lower}}\left( \text{x, y, c} \right) \geq 0:} \\{\text{M}\left( \text{x, y, c} \right) = \text{I}\left( \text{x, y, c} \right)\text{- J}_{\text{lower}}\left( \text{x, y, c} \right)} \\{\cdot \text{When I}\left( \text{x, y, c} \right)\text{- J}_{\text{lower}}\left( \text{x, y, c} \right) < 0:} \\{\text{M}\left( \text{x, y, c} \right) = 0}\end{array} & \text{­­­(11)}\end{matrix}$

Here, M(x, y, c) is the Mie scattering component image. The Miescattering component can be extracted from the image by means of thiscalculation.

In step S1103, the Mie scattering component calculating unit 306 storesthe Mie scattering component image calculated in step S1102 in the Miescattering component data storing unit 314.

By performing processing as described above, the Mie scatteringcomponent in the image can be calculated.

Calculation of Rayleigh Scattering Component

Next, processing (step S506) for calculating the Rayleigh scatteringcomponent, which is performed by the Rayleigh scattering componentcalculating unit 307, will be described with reference to the flowchartin FIG. 12 .

In step S1201, the Rayleigh scattering component calculating unit 307reads the RGB lower-pixel-value corrected image data from the RGBlower-pixel corrected data storing unit 313, reads the Mie scatteringcomponent image from the Mie scattering component data storing unit 314,and reads the input image from the input image data storing unit 204.

In step S1202, the Rayleigh scattering component calculating unit 307subtracts a pixel value for each pixel in the image, in order to obtaina Rayleigh scattering component image. Specifically, the calculation isperformed according to formula (12) below.

$\begin{matrix}\begin{matrix}{\cdot \text{When I}\left( \text{x, y, c} \right)\text{- J}_{\text{RGB}}\left( \text{x, y, c} \right)\text{- M}\left( \text{x, y, c} \right) \geq 0:} \\{\text{R}\left( \text{x, y, c} \right) = \text{I}\left( \text{x, y, c} \right)\text{- J}_{\text{RGB}}\left( \text{x, y, c} \right)\text{- M}\left( \text{x, y, c} \right)} \\{\cdot \text{When I}\left( \text{x, y, c} \right)\text{- J}_{\text{RGB}}\left( \text{x, y, c} \right)\text{- M}\left( \text{x, y, c} \right) < 0:} \\{\text{R}\left( \text{x, y, c} \right) = \text{0}}\end{matrix} & \text{­­­(12)}\end{matrix}$

Here, R(x, y, c) is the Rayleigh scattering component image. TheRayleigh scattering component can be extracted from the image by meansof this calculation.

In step S1203, the Rayleigh scattering component calculating unit 307stores the Rayleigh scattering component image calculated in step S1202in the Rayleigh scattering component data storing unit 315.

By performing processing as described above, the Rayleigh scatteringcomponent in the image can be calculated.

Composing Processing

Next, composing processing (step S507) by the composing unit 308 will bedescribed.

The composing unit 308 calculates a composed image J_(out)(x, y, c)according to formula 13 below.

$\begin{matrix}{\text{J}_{\text{out}}\left( \text{x, y, c} \right) = \text{J}_{\text{RGB}}\left( \text{x, y, c} \right) + \text{m} \cdot \text{M}\left( \text{x, y, c} \right) + \text{r} \cdot \text{R}\left( \text{x, y, c} \right)} & \text{­­­(13)}\end{matrix}$

Here, m is a Mie scattering intensity coefficient, and r is a Rayleighscattering intensity coefficient. In the present embodiment, it isdesirable that the coefficients take a value between zero and one ineach of the first and second parameters, but other values may be used asa matter of course.

According to the present invention, when processing for removing theinfluence of a fine particle component in an image is performed, theeffect of the removal processing can be changed based on objectdetection results, as described above. Accordingly, an image in whichobjects have higher visibility can be obtained.

For example, if a person is set as a target to be detected (object), animage in which the influence of the fine particle component has beenremoved and which is specific to the person can be obtained.

Note that the above-described configurations pertaining to the removalof fine particles may be implemented in an image-capturing apparatustypified by a digital camera. For example, the configurations may beimplemented as an image-capturing mode to be used for performingimage-capturing with a person as a photographic subject in the fog. Inthis case, it suffices to incorporate the image-capturing unit 111 inFIG. 1 as a part of the configuration of the image processing apparatus100.

Note that, with regard to the object extraction processing unit 206, thefine particle removal processing unit, etc., among the above-describedprocessing units, processing may be performed using a pre-trained modelhaving been subjected to machine learning, in place of such units. Inthat case, a plurality of combinations of input data and output data forthe processing unit are prepared as learning data, for example.Knowledge is acquired from the plurality of pieces of learning datathrough machine learning, and a pre-trained model that outputs outputdata corresponding to input data as a result based on the acquiredknowledge is generated. The pre-trained model can be configured by usinga neural network model, for example. Furthermore, the pre-trained modelperforms the processing of the processing unit by operating incooperation with a CPU, a GPU, etc., as a program for performingprocessing equivalent to that by the processing unit. Note that theabove-described pre-trained model may be updated as necessary afterpredetermined processing is performed.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2019-084427, filed Apr. 25, 2019 which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus that processesimage data obtained by image capturing, the image processing apparatuscomprising: a processor; and a memory storing a program which, read andexecuted by the processor, causes the processor to function as: adetecting unit configured to detect an object in an image correspondingto the image data based on the image data; and an image processing unitconfigured to perform a first processing for removing componentcorresponding scattered light caused by fine particles existing in airto a first area where the object has been detected in the image, and toperform a second processing for removing the component to a second areawhere the object has not been detected in the image, the secondprocessing having a different effect for removing the component from thefirst processing.
 2. The apparatus according to claim 1, wherein thesecond processing removes the component to a further extent compared tothe first processing.
 3. The apparatus according to claim 2, furthercomprising a generation unit configured to generate a composed image byselecting image data obtained by the first image processing to the firstarea and by selecting image data obtained by the second image processingto the second area.
 4. The apparatus according to claim 1, furthercomprising a image-capturing unit configured to capture the image dataobtained by the image capturing.
 5. The apparatus according to claim 1,wherein the first image processing and the second image processingperform calculation of a Mie scattering component and calculation of aRayleigh scattering component, and perform the first processing and thesecond processing by generating a composed image in which the calculatedMie scattering component and the Rayleigh scattering component are used.6. The apparatus according to claim 5, wherein the first processing usesa first parameter and the second processing uses a second parameter, thefirst parameter and the second parameter including a Mie scatteringintensity coefficient and a Rayleigh scattering intensity coefficient,the Mie scattering intensity coefficient and the Rayleigh scatteringintensity coefficient respectively indicating a contribution of the Miescattering component and a contribution of the Rayleigh scatteringcomponent in the generation of the composed image, and the Miescattering intensity coefficient and the Rayleigh scattering intensitycoefficient have smaller values in the second parameter than in thefirst parameter.
 7. The apparatus according to claim 1, wherein thefirst area is specified by a user instruction via a user interface.
 8. Amethod of controlling an image processing apparatus that processes imagedata obtained by image capturing, the method comprising: detecting anobject in an image corresponding to the image data based on the imagedata; performing a first processing for removing component correspondingscattered light caused by fine particles existing in air to a first areawhere the object has been detected in the image; and performing a secondprocessing for removing the component to a second area where the objecthas not been detected in the image, the second processing having adifferent effect for removing the component from the first processing.9. A non-transitory computer-readable storage medium storinginstructions which, when read and executed by a computer, causes thecomputer to perform the steps of a method of controlling an imageprocessing apparatus that processes image data obtained by imagecapturing, the method comprising: detecting an object in an imagecorresponding to the image data based on the image data; performing afirst processing for removing component corresponding scattered lightcaused by fine particles existing in air to a first area where theobject has been detected in the image; and performing a secondprocessing for removing the component to a second area where the objecthas not been detected in the image, the second processing having adifferent effect for removing the component from the first processing.